United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
I nilid Stall-, l'atint and Trademark Office 

Address: COMMISSIONER FOR PATENTS 



APPLICATION NO. 



10/815,354 



FILING DATE 



0.V3 1/2004 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. CONFIRMATION NO. 



FXPL-1094US0 



23910 7590 04, 

FLIESLER MEYER LLP 
650 CALIFORNIA STREET 
14TH FLOOR 

SAN FRANCISCO, CA 94108 



PARK, EDWARD 



PAPER NUMBER 



DELIVERY MODE 



Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 



PTOL-90A (Rev. 04/07) 



l/ffflrC? nVrliUli Otfff Iff ids y 


Application No. 

10/815,354 


Applicant(s) 
CHIU ET AL. 


Examiner 

EDWARD PARK 


Art Unit 

2624 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )□ Responsive to communication(s) filed on 4/7/08 . 
2a )□ This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-21 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

Claim(s) is/are allowed. 

6) |EI Claim(s) ±2± is/are rejected. 

7) 0 Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10) ^ The drawing(s) filed on 31 March 2004 is/are: a)^ accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

20 Certified copies of the priority documents have been received in Application No. . 

3.Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attach ment(s) 

1) ^| Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) □ Information Disclosure Statement(s) (PTO/SB/08) 5 ) □ Notice of Informal Patent Application 

Paper No(s)/Mail Date . 6) □ Other: . 



PTOL-T26 d (Rev e 08-06r 



Office Action Summary 



Part of Paper No./Mail Date 20080416 



Application/Control Number: 10/815,354 
Art Unit: 2624 



Page 2 



DETAILED ACTION 

Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1 .17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.1 14, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.1 14. Applicant's submission filed on 4/7/08 has been entered. 

Claim Rejections - 35 USC § 112 

2. In response to applicant's amendment of claim 20, the previous claim rejection is 
withdrawn. 

Claim Rejections - 35 USC §102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed publication in this 
or ;i foreign country, before the invention thereof by the applicant for a patent. 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 2 1 (2) of such treaty in the English language. 

4. Claims 1, 10, 12 are rejected under 35 U.S.C. 102(b) as being anticipated by Jun et al 
(US 2001/0020981 Al). 
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Regarding claim 1, Jun teaches a method for generating a highly condensed visual 
summary of video regions, comprising: 

determining a dominant group in each of a plurality of video segments (Jun: paragraphs 
[0023]-[0024]); 

determining a key frame in each of the video segments (Jun: paragraphs: [0023]-[0024]); 

defining a germ associated with each dominant group in each of the video segments (Jun: 
paragraph [0051], region of interest is read as "a germ"); 

laying out the germs on a canvas, each germ associated with a support; and filling in the 
space of the canvas between the germs (Jun: figures: 13a, 13b, 17, paragraphs [0074]-[0076], 
Examiner notes that it can be seen in figure 13a, for example, that the germ is a region of 
interest, and the claim limitation filling in the space of the canvas between the germs only brings 
in the limitation of occupying the canvas with any additional image that is not part of the region 
of interest, which can be seen in figure 13a in the Fsk. The region of interest "germ" can be the 
head of the person; and any other background imagery that is not part of the region of interest 
which is seen in the figure will meet the limitation of the claim). 

Regarding claim 10, Jun teaches a method for generating a highly condensed visual 
summary of video regions, comprising: 

determining a germ in each of a plurality of images, the germ containing a region of interest 
(Jun: paragraph [0051], region of interest is read as "a germ"); 

laying out the germs on a canvas, each germ associated with a support; and filling in the 
space of the canvas between the germs with one or more parts of the image from the support 
(Jun: figures: 13a, 13b, 17, paragraphs [0074]-[0076], Examiner notes that it can be seen in 
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figure 13 a, for example, that the germ is a region of interest, and the claim limitation filling in 
the space of the canvas between the germs only brings in the limitation of occupying the canvas 
with any additional image that is not part of the region of interest, which can be seen in figure 
13a in the Fsk. The region of interest "germ" can be the head of the person; and any other 
background imagery that is not part of the region of interest which is seen in the figure will meet 
the limitation of the claim). 

Regarding claim 12, Jun teaches receiving user input, the user input associated with a 
part of an image (Jun: paragraph [0077]). 

Claim Rejections - 35 USC §103 

5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

6. Claims 2-6, 13-15, 20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Jun 
et al (US 2001/0020981 Al) in view of Uchihashi (ACM Multimedia: "Video Manga: 
Generating Semantically Meaningful Video Summaries"). 

Regarding claim 2, Jun discloses all elements as mentioned above in claim 1 . Jun does 
not teach determining a group within each of the plurality of video segments having the largest 
volume. 
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Uchihashi teaches determining a group within each of the plurality of video segments 
having the largest 3-D volume (Uchihashi: section 4.2, length of the segment is scored). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Jun reference to determine a group having the largest volume as taught by 
Uchihashi, in order to "calculate an importance score for each segment based on its rarity and 
duration" since "a segment is deemed less important if it is short or very similar to other 
segments" (Uchihashi: section 4.2). 

Regarding claims 3, 4, and 20, Jun discloses all elements as mentioned above in claim 1. 
Jun does not teach defining a two dimensional shape that encompasses the projection of the 
dominant group onto the key frame; wherein the two dimensional shape is a rectangle; and using 
an algorithm to determine a region of interest of an image. 

Uchihashi teaches defining a two dimensional shape that encompasses the projection of 
the dominant group onto the key frame (Uchihashi: figure 2; section 4.4) and wherein the two 
dimensional shape is a rectangle (Uchihashi: figure 2; section 4.4). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Jun reference to define a two dimensional shape that is a rectangle as taught 
by Uchihashi, in order to "form a pictorial abstract of the video sequence" where a "sequence of 
frames .... fills space efficiently and represents the original video sequence well" (Uchihashi: 
section 4.4). 

Uchihashi further teaches using an algorithm to determine a region of interest of an image 
(Uchihashi: figure 4.2). 
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It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Jun with Uchihashi combination as mentioned above to determine a region 
of interest of an image as taught by Uchihashi, "to select appropriate keyframes for a compact 
pictorial summary" (Uchihashi: section 4.2). 

Regarding claims 5 and 6, Jun with Uchihashi discloses all elements as mentioned above 
in claim 3. Jun with Uchihashi as mentioned in claim 3, does not teach determining a scale 
factor to be applied to every germ such that the germs are scaled to the maximum size that fits 
into the canvas and placing the germs in rows, wherein each row has a height according to the 
longest germ in the particular row. 

Uchihashi further teaches determining a scale factor to be applied to every germ such that 
the germs are scaled to the maximum size that fits into the canvas (Uchihashi: section 4.3, 4.4) 
and placing the germs in rows, wherein each row has a height according to the longest germ in 
the particular row (Uchihashi: figure 2). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Jun with Uchihashi combination to place the germs in a row as taught by 
Uchihashi, to "fill space efficiently and represent the original video sequence well" (Uchihashi: 
section 4.2). 

Regarding claim 13, Jun discloses all elements as mentioned above in claim 10. Jun does 
not disclose using an algorithm to determine the regions of interest of an image based on one or 
more methods selected from the group consisting of a general image analysis algorithm, a face- 
detection algorithm, and object detection algorithms and user input. 
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Uchihashi teaches using an algorithm to determine the regions of interest of an image 
based on one or more methods selected from the group consisting of a general image analysis 
algorithm (see section 4.2, segment is scored and weighted), a face-detection algorithm, and 
object detection algorithms and user input. 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Jun reference to determine the regions of interest with a general image 
analysis algorithm as taught by Uchihashi, "to select appropriate keyframes for a compact 
pictorial summary" (Uchihashi: section 4.2). 

Regarding claims 14 and 15, Jun reference discloses all elements as mentioned above in 
claim 10. Jun reference as mentioned in claim 10, does not teach determining a scale factor to be 
applied to every germ such that the germs are scaled to the maximum size that fits into the 
canvas and placing the germs in rows, wherein each row has a height according to the longest 
germ in the particular row. 

Uchihashi further teaches determining a scale factor to be applied to every germ such that 
the germs are scaled to the maximum size that fits into the canvas (Uchihashi: section 4.3, 4.4) 
and placing the germs in rows, wherein each row has a height according to the longest germ in 
the particular row (Uchihashi: figure 2). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Jun reference to place the germs in a row as taught by Uchihashi, to "fill 
space efficiently and represent the original video sequence well" (Uchihashi: section 4.2). 
7. Claims 7-9, 16-18 are rejected under 35 U.S.C. 103(a) as being unpatentable over Jun et 
al (US 2001/0020981 Al) in view of Hirata (US 6,922,485 B2). 
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Regarding claims 7-9, Jun discloses all elements as mentioned above in claim 1 . Jun 
does not teach assigning a pixel value of each point in the canvas to the same pixel value in the 
support associated with the germ closest to each point; wherein if the germ closest to the point 
does not have a support that includes the point, the point is assigned the pixel value of the closest 
germ with a support that includes the point; wherein the point is assigned a background value if 
no support includes the point. 

Hirata teaches assigning a pixel value of each point in the canvas to the same pixel value 
in the support associated with the germ closest to each point (see col. 8, lines 62-67; col. 9, lines 
1-14); wherein if the germ closest to the point docs not have a support that includes the point, the 
point is assigned the pixel value of the closest germ with a support that includes the point (see 
col. 8, lines 62-67; col. 9, lines 1-14); wherein the point is assigned a background value if no 
support includes the point (see col. 8, lines 62-67; col. 9, lines 1-14). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Jun reference to assign pixel value to a germ that is closest as taught by 
Hirata, to create an aesthetically/visually pleasing image to the user by removing white spaces 
between germs. 

Regarding claims 16-18, Jun discloses all elements as mentioned above in claim 10. Jun 
does not teach assigning a pixel value of each point in the canvas to the same pixel value in the 
support associated with the germ closest to each point; wherein if the germ closest to the point 
does not have a support that includes the point, the point is assigned the pixel value of the closest 
germ with a support that includes the point; wherein the point is assigned a background value if 
no support includes the point. 
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Hirata teaches assigning a pixel value of each point in the canvas to the same pixel value 
in the support associated with the germ closest to each point (see col. 8, lines 62-67; col. 9, lines 
1-14); wherein if the germ closest to the point does not have a support that includes the point, the 
point is assigned the pixel value of the closest germ with a support that includes the point (see 
col. 8, lines 62-67; col. 9, lines 1-14); wherein the point is assigned a background value if no 
support includes the point (see col. 8, lines 62-67; col. 9, lines 1-14). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Jun reference to assign pixel value to a germ that is closest as taught by 
Hirata, to create an aesthetically/visually pleasing image to the user by removing white spaces 
between germs. 

8. Claims 11, 19 is rejected under 35 U.S.C. 103(a) as being unpatentable over Jun et al 
(US 2001/0020981 Al) in view of Li et al (US 7,035,435 B2). 

Regarding claim 11, Jun discloses all elements as mentioned above in claim 10. Jun does 
not teach detecting a face in each of the plurality of images. 

Li teaches detecting a face in each of the plurality of images (Li: col. 7, lines 33-51). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Jun reference to detect a face as taught by Li, in order to determine the 
importance of a frame since "a human face will be more informative than, for example, a 
landscape frame" (Li: col. 7, lines 33-51). 

Regarding claim 19, Jun discloses all elements as mentioned above in claim 1. Jun does 
not teach detecting a face in each of the plurality of images. 

Li teaches detecting a face in each of the plurality of images (Li: col. 7, lines 33-51). 
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It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Jun reference to detect a face as taught by Li, in order to determine the 
importance of a frame since "a human face will be more informative than, for example, a 
landscape frame" (Li: col. 7, lines 33-51). 

9. Claim 21 is rejected under 35 U.S.C. 103(a) as being unpatentable over Jun et al (US 
2001/0020981 Al) in view of Lin et al (US 6,307,964 Bl). 

Regarding claim 21, Jun discloses all elements as mentioned above in claim 1. Jun does 
not disclose using a Voronoi algorithm to determine the shape of the support to be placed on the 
canvas. 

Lin, in the same field of endeavor, teaches using a Voronoi algorithm to determine the 
shape of the support to be placed on the canvas (see col. 2, lines 41-46, representing a shape of 
an object in an image). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Jun reference to utilize a Voronoi algorithm as taught by Lin, in order 
increase the "effectiveness of a shape descriptor" by being "able to capture the structural 
information of the shape, yet be robust to noise within the boundary [to] yield the most favorable 
result" (see col. 2 lines 32-40). 

Response to Arguments 

10. In regards to the applicant's general remarks on pages 7 and 8, the applicant states several 
statements made by the examiner. Examiner does not recall making such statements or 
agreements as applicant has stated. Examiner notes that the interview summary clearly states 
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what was discussed on record, and no conclusion or agreement was reached in any of the claims. 
Examiner reminds applicant of MPEP 713.04 [R-3], 37 CFR 1.2, "No attention will be paid to 
any alleged oral promise, stipulation, or understanding in relation to which there is disagreement 
or doubt." Furthermore, applicant's request for Supervisory Examiner Brian Werner to write a 
separate summary of the interview is denied since an interview summary was already mailed and 
was signed by Supervisory Examiner Brian Werner that is dated on 10/3/07. Examiner notes that 
the remarks made on pages 7 and 8 do not carry any weight since there was no agreement 
reached during the interview. 

Applicant's arguments filed 4/7/08, in regards to claim 1, have been fully considered but 
they are not persuasive. Applicant argues that the Jun reference does not disclose the "filling in 
the space of the canvas between the germs" limitation of claim 1 . This argument is not 
considered persuasive since it can be clearly seen in figure 13a and paragraphs 0074-0075 that 
the synthetic key frame FSK presents an image by combining key frames or key regions 
representing the entire content of the scene. It can be seen in figure 13a, for example, that the 
germ is a region of interest "germ", and the claim limitation filling in the space of the canvas 
between the germs only brings in the limitation of occupying the canvas with any additional 
image that is not part of the region of interest, which can be seen in figure 13a in the Fsk. The 
region of interest can be the head of the person; and any other background imagery that is not 
part of the region of interest which is seen in the figure will meet the limitation of the claim. 
Since the claim limitation only calls for filling in the space of the canvas between the germs is 
equivalent to having any information, whether it is background information between each region 
of interest. Furthermore, the new claim limitation, does not limit the scope of what the space 
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between the germs is being filled with. Therefore, examiner is reading the limitation of filling in 
the space of the canvas between the germs as being equivalent to having any support associated 
with a germ. This is due to the fact the support surrounds the germ, which is between two or 
more germs. Therefore, the limitation of this claim is met by the reference and the amendment 
does not, if any, further limit the scope and limitation. 

In regards to claim 10, applicant argues that the Jun reference does not disclose "filling in 
the space between the germs". This argument is similar to the argument of claim 1 . Examiner 
notes to see arguments for claim 1 since it is identical to claim 10. 

In regards to claim 12, applicant argues that the claim is patentable due to the 
dependency of claim 10. This argument is not considered persuasive since the rejection of claim 
10 stands and the rejection and arguments can be seen above. 

In regards to claim 2, applicant argues that the Uchihashi reference does not teach a 3-D 
volume, but a 1-D volume. This argument is not considered persuasive and technologically 
incorrect in reference to the claim. How is volume defined in one dimension? The Uchihashi 
reference teaches determining the longest segment and scores the segment. Examiner notes that 
the applicant argues limitations that are not in claim 2, and therefore the examiner reiterates the 
limitations of claim 2. Claim 2 calls for "determining a group within each of the plurality of 
video segments having the largest 3-D volume". Basically, the claim calls for the determination 
of a group within a segment of video frames that has the longest running time. Applicant argues 
irrelevant issues that do not pertain to claim 2 and tries to bring in limitations from the 
specification into the claim language which is both incorrect and does not make technological 
sense. 
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In response to applicant's argument that the references fail to show certain features of 
applicant's invention, it is noted that the features upon which applicant relies (i.e., project the 
dominant group onto the key frame) are not recited in the rejected claim(s). Although the claims 
are interpreted in light of the specification, limitations from the specification are not read into the 
claims. See In re Van Geuns, 988 F.2d 1 181, 26 USPQ2d 1057 (Fed. Cir. 1993). 

In regards to claims 2-6, 13-15, 20, applicant argues that the claim is patentable since it 
depends from claims 1 and 10 respectively. This argument is not persuasive since the rejection 
of claims 1 and 10 stand and the rejection and arguments can be seen above. 

Applicant's arguments with respect to claims 7-9, 16-18 have been considered but are 
moot in view of the new ground(s) of rejection. 

In regards to claims 11, 19, applicant argues that the claim is patentable since it depends 
from claims 10 and 1 respectively. This argument is not persuasive since the rejection of claims 
10 and 1 stand and the rejection and arguments can be seen above. 

In regards to claim 21, applicant argues that the claim Lin reference does not disclose the 
contour as being part of the output. This argument is not considered persuasive since the claim 
limitation does not call for the applicant's limitation in the arguments. In response to applicant's 
argument that the references fail to show certain features of applicant's invention, it is noted that 
the features upon which applicant relies (i.e., "contours are part of the output") are not recited in 
the rejected claim(s). Although the claims are interpreted in light of the specification, limitations 
from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1 181, 26 
USPQ2d 1057 (Fed. Cir. 1993). Furthermore, the applicant argues that the Lin's reference, 
Voronoi function is different from the applicant's Voronoi algorithm. This argument is not 
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considered persuasive since the Voronoi function is used in many different aspects in 
mathematics and image analysis; and in respect to the claim, again the applicant is reading in 
limitations from the specification into the claims, which is not proper. 

Conclusion 

1 1 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to EDWARD PARK whose telephone number is (571)270-1576. 
The examiner can normally be reached on M-F 10:30 - 20:00, (EST). 

If attempts to reach the examiner by telephone arc unsuccessful, the examiner's 
supervisor, Vikkram Bali can be reached on (571) 272-7415. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Edward Park 
Examiner 
Art Unit 2624 

/Edward Park/ 
Examiner, Art Unit 2624 
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